Replacing Shell Scipt with Python

Mon 11 May 17:03 IST

Hi,

This is the first entry in my intern diary, I'm interning at Continuum Analytics a software company based in Austin, USA. I was supposed to be at Austin office but due to visa issues I couldn't travel to US. Hence I'm working remotely.

I've been give a project a automatically build Python packages from the Python Package Index for Continuum's Python agnostic cross platform package manager conda. I started my intern last monday and last week:

• I learned shell scripting and wrote little shell scripts to automatically create conda recipes using conda-skeleton.
• learned about some of its internals of conda build and fixed a little bug
• Wrote a script to get download statistics from PyPI using vanity
• got an AWS instance and created recipes of some 33 thousand packages from 59 thousand packages on PyPI of which the recipe creation process failed for some seven thousand packages. In the process also learned about screen so that I can run processes on remote server with a fear of disconnection from my end.

I have not analyzed every failure but it appears that most of the recipes failures, conda-skeleton was not at fault. A lot of packages were non-existent, or there was something wrong in the setup.py. Today I set to analyze these failures and listed some of the typical errors. I wanted to classify package on basis of the error and really want to figure out if there is something wrong with the package or with conda-skeleton, for that I had to make additional modifications to the shell script but I was kinda disgusted from its current state. Also I was overwhelemed with the sheer number packages, so I first wrote a script to randomly select n number of packages from a list of packages and then rewrote the shell script in python. In the process I learned the argparse and the subprocess. I was impressed by the ease of using argparse.